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Overview 


□)  Defective  software  is  not  secure 
The  Team  Software  Process 
TSP  and  secure  systems  development 
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Defective  Software  is  not  Secure 

Common  software  defects  are  a  principal  cause  of 
software  security  incidents. 

•  Over  90%  of  software  security  incidents  are  due  to 
attackers  exploiting  known  software  defect  types.1 

•  Analysis  of  forty-five  e-business  applications  showed 
that  70%  of  the  security  defects  were  design  defects.2 

Conclusion:  there  is  no  such  thing  as  a  poor  quality  secure 
system. 


CERT/CC 

"The  Security  of  Applications:  Not  All  Are  Created  Equal",  by  Andrew  Jacquith. 
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Security  Design  Defects 

Examples 

•  Failure  to  authorize  and  authenticate  users 

•  Failure  to  validate  user  input 

•  Failure  to  encrypt  and/or  protect  sensitive  data 

Everyday  software  “bugs”  are  also  a  major  risk. 

For  example,  a  buffer  overflow  can  cause  system  failure, 
or  allow  a  hacker  to  take  control  of  your  system. 

Many  common  defect  types  can  produce  a  buffer  overflow. 

•  declaration  error 

•  logic  errors  in  loop  control  or  conditional  expression 

•  failure  to  validate  input 

•  interface  specification  error 
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The  Software  Quality  Problem 

Software  quality  is  highly  variable  and  generally  poor  in 
non-mission  critical  systems. 

Widely-used  operating  systems  and  applications  software 
are  known  to  have  more  than  2  defects  per  KSLOC,  or 
2000+  defects  per  million  SLOC. 

If  only  5%  of  these  defects  are  potential  security  concerns, 
there  are  100  vulnerabilities  per  million  SLOC. 
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Software  Practice  and  Quality 

Software  is  the  only  modern  technology  that  ignores 
quality  until  test.  Typically,  software  engineers 

•  do  not  plan  their  own  work 

•  race  through  requirements  and  design 

•  do  the  design  while  coding 

These  practices  introduce  volumes  of  defects. 

•  Experienced  engineers  inject  a  defect 
every  7  to  10  lines  of  code. 

•  For  even  moderate-sized  systems, 
this  amounts  to  thousands  of  defects. 

•  Most  of  these  defects  must  be 
found  in  test. 

•  This  usually  takes  about  half  of  the 
development  schedule. 
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The  Problem  with  Testing 


Security  Attack 


Resource 
contention 


Hardware 

failure 


Operator 

error 


Data  error 


Safe  region  =  tested 
(shaded) 

Unsafe  region  =  untested 
(unshaded) 
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Principles  of  Software  Engineering 

Current  software  practice  violates  well  understood 
principles  of  software  engineering. 

Examples  of  software  engineering  principles 

•  the  need  for  accurate  plans 

•  the  importance  of  detailed,  verifiable  designs 

•  early  defect  removal 

•  effective  inspections 

•  focus  on  quality  throughout 

Why  are  these  principles  not  applied? 
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Principles  Are  Not  Enough 

Principles  are  easy  to  understand,  but  harder  to  follow. 

To  apply  these  principles  in  practice  you  also  need 

•  a  supportive  infrastructure  and  environment 

•  an  operational  process  (rules  and  steps)  to  put  the 
principles  into  practice 

•  a  measurement  system  to  manage  and  control  the 
result 

Software  engineers  also  need  to  be  convinced  of  the 
benefits  of  disciplined  software  engineering  methods. 
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Design  Principles  for  Secure  Applications 

The  principles  for  producing  secure  applications  are  also 
well  known  and  easy  to  understand. 

Examples 

•  Authorize  and  authenticate  all  users 

•  Mistrust  all  user  input 

•  Encrypt  sensitive  data  from  login  to  logout 

•  Protect  persistent  data 
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What  is  the  Issue? 

Lack  of  understanding  is  not  the  issue. 

A  lack  of  data  is  not  the  issue.  A  vast  amount  of  incident- 
specific  and  system-specific  data  exists. 

Training  is  not  the  issue.  Training  is  available  for 

•  writing  secure  applications 

•  network  administration 

Support  is  not  the  issue.  There  are  emergency  response 
centers  (including  the  CERT/CC),  guidelines,  checklists, 
best  practices,  etc. 
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More  Is  Needed 

With  all  the  information  and  resources  available,  why  is 
security  still  an  issue? 

Maybe  there  is  a  need  for  more. 

•  an  environment  that  fosters  good  practice 

•  operational  processes  based  on  engineering  principles 

•  disciplined  practitioners  that  adhere  to  these  principles 

•  predictive  measures 
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Overview 


Defective  software  is  not  secure 
^  The  Team  Software  Process 

TSP  and  secure  systems  development 
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The  Team  Software  Process  -1 

The  Team  Software  Process  (TSP)  is  an  operational 
process  designed  to  support  well-established  principles  of 
software  engineering. 

The  principal  objectives  of  the  TSP  are 

•  help  software  engineering  teams  build  quality  products 
within  cost  and  schedule  constraints 

•  build  teams  quickly  and  reliably 

•  optimize  team  performance  throughout  a  project 
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The  Team  Software  Process  -2 

TSP  incorporates  best  practices  of  software  engineering  in 
a  single  integrated  package,  e.g. 

•  team  project  management 

•  product  quality  management 

•  process  management 

•  risk  management 

•  software  metrics 

With  TSP,  software  teams 

•  build  detailed,  accurate  plans 

•  manage  and  track  their  commitments  to  within  +/- 10% 

•  produce  near  defect-free  software  with  typically  less 
than  0.1  defects/KSLOC 
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Personal  Software  Process 

To  use  the  TSP,  software  developers  must  first  be  trained 
in  the  Personal  Software  Process  (PSP). 

The  PSP  provides  software  developers  with  the  skills  and 
self-convincing  evidence  of  the  benefits  of  software 
engineering  practice. 

In  using  the  PSP,  software  developers 

•  follow  a  defined  and  measured  personal  process 

•  plan  every  job  before  they  do  it 

•  gather  time,  size,  and  defect  data  as  they  work 

•  use  these  data  to  manage  their  personal  work  and 
ensure  the  quality  of  the  products  they  produce 
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Software  Quality  with  TSP 

The  quality  management  practices  in  TSP  and  PSP 
dramatically  reduce  the  product  defect  content. 

The  following  results  are  drawn  from 

•  PSP  training  data  on  298  software  developers 

•  TSP  data  from  18  projects  in  four  organizations 

The  results  show  the  effect  of  TSP  and  PSP  on 

•  process  quality 

•  design  time 

•  product  quality 

•  system  test  duration 
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Defects  Removed  Before  Compile 
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Process  Quality  Results 


Pre-Compile  Defect  Yield 


Mean  Yield 
PSP  Level  Mean 
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PSP  Design  Time  Results 


Time  Invested  Per  (New  and  Changed)  Line  of  Code 


■Q  Design 

-♦ -  Code 

B -  Compile 

■o—  Test 
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Mean  Number  of  Defects  Per 
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PSP  Product  Quality 


Defects  Per  KLOC  Removed  in  Compile  and  Test 
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PSP  Productivity  Results 


Lines  of  (New  and  Changed)  Code 
Produced  Per  Hour  of  Total  Development  Time 


Mean  LOC/Hour 
PSP  Level  Average 
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TSP  Quality  Improvement  -1 


#  of  defects 
detected 


Software  Size 


2.36X  more 
Sloe  count 


Release  #  6  Release  #  7 


Release  #  8  Release  #  9 
PSP/TSP- 
trained 


©2002  by  Carnegie  Mellon  University 


Version  1.0 


TSP  for  Secure  Systems  2002.03.1523 


Carnegie  Mellon 

Software  Engineering  Institute 


TSP  Quality  Improvement  -2 


System 
Test  time 


(Boeing  Pilot  #1) 


32  days 


41  days 


28  days 


2.36X  more 
Sloe  count 


94%  less  time 


4  days 


Release  #  6  Release  #  7 
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TSP  Project  Results 

Plan  Actual 

Size  Estimate  110,000  LOC  89,995  LOC 

Effort  Estimate  16,000  hours  14,711  hours 

Schedule  77  weeks  71  weeks 

Product  Quality  (Defects/KLOC  removed  in  phase) 

•  Integration  1.0  0.2 

•  System  Test  0.1  0.4 

•  Field  Trial  0.0  0.02 

Benefits 

•  Quality  levels  improved  20  times  over  prior  projects. 

•  Actual  effort  and  schedule  were  within  8%  of  plan  (early). 
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TSP  Results  Summary  -1 


Category 

Without  TSP 

With  TSP 

Average  schedule 
deviation  -  range 

27%  to  112% 

-8%  to  5% 

Average  effort  deviation 
-  range 

17%  to  85% 

-8%  to  -4% 

Acceptance  test  product 
quality  (defects/KLOC) 

0.1*  to  0.7 

0.02  to  0.1 

System  test  savings  (cost  to 
system  test  1000  LOC) 

1  to  5  days 

0.1  to  1  days 

Number  of  post-release 
defects  per  KLOC 

0.2  to  1  + 

Oto  0.1 

*  This  data  (.1  defects/KLOC  in  acceptance  test)  is  from  a  CMM  Level  5  organization 
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TSP  Results  Summary  -2 


Average  Effort  Deviation  -  Range 

120%  -| - 

100% 

80%  - 
60%  - 
40%  - 

20%  -  _ 

0% - , - - 

-20%  J - 

Pre  TSP/PSP  With  TSP/PSP 


Defects/KLOC  in  Acceptance  Test  -  Range 

0.9  - 

0.7  - 
0.6  - 
0.5  - 
0.4  - 
0.3  - 
0.2  - 


Pre  TSP/PSP  With  TSP/PSP 


Average  Schedule  Deviation  -  Range 

160%  -| - 

140% 

120%  - 
100%  - 
80%  - 
60%  - 
40%  - 
20%  - 

0% - j - L  i>- 

-20%  J - 

Pre  TSP/PSP  With  TSP/PSP 


Post-Release  Defects/KLOC  -  Range 
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Overview 


Defective  software  is  not  secure 
The  Team  Software  Process 
^  TSP  and  secure  systems  development 
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TSP  and  Secure  Systems  -1 

The  TSP  provides  a  framework,  a  set  of  processes,  and 
disciplined  methods  for  producing  quality  software. 

Software  produced  with  TSP  has  one  or  two  orders  of 
magnitude  fewer  defects  than  current  practice. 

•  0.02  defects/KSLOC  vs.  2  defects/KSLOC 

•  20  defects  per  MSLOC  vs.  2000  defects  per  MSLOC 

If  5%  of  the  defects  are  potential  security  holes,  with  TSP 
there  would  be  1  vulnerability  per  million  SLOC. 
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TSP  and  Secure  Systems  -2 

TSP  also  address  the  need  for 

•  professional  behavior 

•  supportive  environment 

•  sound  software  engineering  practice 

•  operational  processes 

•  software  metrics 

With  tailoring,  TSP  could  be  even  more  effective  in  this 
development  domain. 
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TSP  for  Secure  Systems  -1 

TSP  for  Secure  Systems  is  an  applied  research  effort  to 
enhance  TSP  for  the  secure  systems  domain. 

Using  design  principles  for  secure  applications,  the  TSP 
could  be  extended  to  incorporate 

•  secure  design  process 

•  secure  implementation  process 

•  secure  review  and  inspection  process 

•  secure  test  process 

•  security-related  predictive  measures 
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TSP  for  Secure  Systems  -2 

The  goal  of  this  effort  is  to  develop  a  process  that 

•  supports  secure  systems  development  practices 

•  predicts  the  likelihood  of  latent  security  defects 

•  can  be  dynamically  tailored  to  respond  to  new  threats 

The  TSP  for  secure  systems  project  is  planned  for  FY03 
and  will  be  a  collaborative  effort  involving 

•  industry  and  government  partners 

•  SEI/NSS  program 

•  SEI/TSP  initiative 
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TSP  Secure  Systems  Pilot 

A  key  element  of  the  TSP  for  Secure  Systems  project  will 
be  pilot  projects. 

The  pilot  project  will  help  in  developing  the  specific 
technical  solutions. 

•  design  and  implementation  practices 

•  review  methods  and  checklists 

•  tools 

We  are  currently  seeking  organizations  that  are  interested 
in  participating. 
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Summary 

TSP  helps  organizations  establish  a  mature  and 
disciplined  software  engineering  practice  that 

•  improves  cost  and  schedule  predictability 

•  reduces  time  to  market 

•  produces  high-quality,  reliable  software,  with  fewer 
security-related  defects 

The  TSP  for  secure  systems  effort  will  build  on  these 
capabilities  to  create  a  mature  process,  with  specific 
features  for  building  secure  systems. 
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For  More  Information 

Visit  the  Software  Engineering  Institute  web  site  at 
www.sei.cmu.edu 


Visit  the  TSP  web  site  at 
www.sei.cmu.edu/tsp 

Contact  SEI  Customer  Relations  at  412-268-5800 
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